Inverse Reinforcement Learning for Trajectory Imitation Using Static Output Feedback Control.

This article studies the trajectory imitation control problem of linear systems suffering external disturbances and develops a data-driven static output feedback (OPFB) control-based inverse reinforcement learning (RL) approach. An Expert-Learner structure is considered where the learner aims to imitate expert's trajectory. Using only measured expert's and learner's own input and output data, the learner computes the policy of the expert by reconstructing its unknown value function weights and thus, imitates its optimally operating trajectory. Three static OPFB inverse RL algorithms are proposed. The first algorithm is a model-based scheme and serves as basis. The second algorithm is a data-driven method using input-state data. The third algorithm is a data-driven method using only input-output data. The stability, convergence, optimality, and robustness are well analyzed. Finally, simulation experiments are conducted to verify the proposed algorithms.

Berberine hydrochloride-loaded dung beetle chitosan/sodium alginate microspheres ameliorate DSS-induced colitis and regulate gut microorganisms in mice.

Berberine hydrochloride (BH) has long been known for its therapeutic efficacy. In the present study, we aimed to treat mice with colitis using dung beetle chitosan (DCS) -transported BH. To achieve this, BH-loaded DCS/sodium alginate microspheres (SA-DCS-BH) were prepared. The SA-DCS-BH was characterized using SEM, DLS, FT-IR, and XRD, then was used for administration and anti-inflammatory examination in mice. SEM and DLS confirmed the surface morphology of the microspheres, and the particle size was relatively uniform. FT-IR and XRD results confirmed that BH was successfully loaded. In vitro and in vivo studies showed that SA-DCS-BH had slow-release ability. After treatment with SA-DCS-BH, DAI was significantly reduced, colon weight and length increased, spleen length and weight reduced, concentrations of pro-inflammatory cytokines in colonic tissues were reduced, and gut microbiota species abundance was modulated. In addition, this study found a correlation between specific microbes and colitis indicators, Muribaculaceae showed sequential growth after receiving BH, SA-CS-BH, and SA-DCS-BH treatments, respectively. It was concluded that SA-DCS-BH effectively delivered the BH to the intestine with slow-release ability and exhibited anti-inflammatory effects by immune response. Compared to commercial chitosan, DCS has potential for modulating intestinal microorganisms and more suitable carrier for intestinal drug delivery systems.

Inverse Q-Learning Using Input-Output Data.

This article addresses the problem of learning the objective function of linear discrete-time systems that use static output-feedback (OPFB) control by designing inverse reinforcement learning (RL) algorithms. Most of the existing inverse RL methods require the availability of states and state-feedback control from the expert or demonstrated system. In contrast, this article considers inverse RL in a more general case where the demonstrated system uses static OPFB control with only input-output measurements available. We first develop a model-based inverse RL algorithm to reconstruct an input-output objective function of a demonstrated discrete-time system using its system dynamics and the OPFB gain. This objective function infers the demonstrations and OPFB gain of the demonstrated system. Then, an input-output Q -function is built for the inverse RL problem upon the state reconstruction technique. Given demonstrated inputs and outputs, a data-driven inverse Q -learning algorithm reconstructs the objective function without the knowledge of the demonstrated system dynamics or the OPFB gain. This algorithm yields unbiased solutions even though exploration noises exist. Convergence properties and the nonunique solution nature of the proposed algorithms are studied. Numerical simulation examples verify the effectiveness of the proposed methods.

Inverse Reinforcement Learning for Adversarial Apprentice Games.

This article proposes new inverse reinforcement learning (RL) algorithms to solve our defined Adversarial Apprentice Games for nonlinear learner and expert systems. The games are solved by extracting the unknown cost function of an expert by a learner using demonstrated expert's behaviors. We first develop a model-based inverse RL algorithm that consists of two learning stages: an optimal control learning and a second learning based on inverse optimal control. This algorithm also clarifies the relationships between inverse RL and inverse optimal control. Then, we propose a new model-free integral inverse RL algorithm to reconstruct the unknown expert cost function. The model-free algorithm only needs online demonstration of the expert and learner's trajectory data without knowing system dynamics of either the learner or the expert. These two algorithms are further implemented using neural networks (NNs). In Adversarial Apprentice Games, the learner and the expert are allowed to suffer from different adversarial attacks in the learning process. A two-player zero-sum game is formulated for each of these two agents and is solved as a subproblem for the learner in inverse RL. Furthermore, it is shown that the cost functions that the learner learns to mimic the expert's behavior are stabilizing and not unique. Finally, simulations and comparisons show the effectiveness and the superiority of the proposed algorithms.

Data-Driven H∞ Optimal Output Feedback Control for Linear Discrete-Time Systems Based on Off-Policy Q-Learning.

This article develops two novel output feedback (OPFB) Q -learning algorithms, on-policy Q -learning and off-policy Q -learning, to solve H∞ static OPFB control problem of linear discrete-time (DT) systems. The primary contribution of the proposed algorithms lies in a newly developed OPFB control algorithm form for completely unknown systems. Under the premise of satisfying disturbance attenuation conditions, the conditions for the existence of the optimal OPFB solution are given. The convergence of the proposed Q -learning methods, and the difference and equivalence of two algorithms are rigorously proven. Moreover, considering the effects brought by probing noise for the persistence of excitation (PE), the proposed off-policy Q -learning method has the advantage of being immune to probing noise and avoiding biasedness of solution. Simulation results are presented to verify the effectiveness of the proposed approaches.

Inverse Reinforcement Q-Learning Through Expert Imitation for Discrete-Time Systems.

In inverse reinforcement learning (RL), there are two agents. An expert target agent has a performance cost function and exhibits control and state behaviors to a learner. The learner agent does not know the expert's performance cost function but seeks to reconstruct it by observing the expert's behaviors and tries to imitate these behaviors optimally by its own response. In this article, we formulate an imitation problem where the optimal performance intent of a discrete-time (DT) expert target agent is unknown to a DT Learner agent. Using only the observed expert's behavior trajectory, the learner seeks to determine a cost function that yields the same optimal feedback gain as the expert's, and thus, imitates the optimal response of the expert. We develop an inverse RL approach with a new scheme to solve the behavior imitation problem. The approach consists of a cost function update based on an extension of RL policy iteration and inverse optimal control, and a control policy update based on optimal control. Then, under this scheme, we develop an inverse reinforcement Q-learning algorithm, which is an extension of RL Q-learning. This algorithm does not require any knowledge of agent dynamics. Proofs of stability, convergence, and optimality are given. A key property about the nonunique solution is also shown. Finally, simulation experiments are presented to show the effectiveness of the new approach.

Distributed Minmax Strategy for Multiplayer Games: Stability, Robustness, and Algorithms.

This article studies a distributed minmax strategy for multiplayer games and develops reinforcement learning (RL) algorithms to solve it. The proposed minmax strategy is distributed, in the sense that it finds each player's optimal control policy without knowing all the other players' policies. Each player obtains its distributed control policy by solving a distributed algebraic Riccati equation in a multiplayer noncooperative game. This policy is found against the worst policies of all the other players. We guarantee the existence of distributed minmax solutions and study their L2 and asymptotic stabilities. Under mild conditions, the resulting minmax control policies are shown to improve robust gain and phase margins of multiplayer systems compared to the standard linear-quadratic regulator controller. Distributed minmax solutions are found using both model-based policy iteration and data-driven off-policy RL algorithms. Simulation examples verify the proposed formulation and its computational efficiency over the nondistributed Nash solutions.

Robust Inverse Q-Learning for Continuous-Time Linear Systems in Adversarial Environments.

This article proposes robust inverse Q -learning algorithms for a learner to mimic an expert's states and control inputs in the imitation learning problem. These two agents have different adversarial disturbances. To do the imitation, the learner must reconstruct the unknown expert cost function. The learner only observes the expert's control inputs and uses inverse Q -learning algorithms to reconstruct the unknown expert cost function. The inverse Q -learning algorithms are robust in that they are independent of the system model and allow for the different cost function parameters and disturbances between two agents. We first propose an offline inverse Q -learning algorithm which consists of two iterative learning loops: 1) an inner Q -learning iteration loop and 2) an outer iteration loop based on inverse optimal control. Then, based on this offline algorithm, we further develop an online inverse Q -learning algorithm such that the learner mimics the expert behaviors online with the real-time observation of the expert control inputs. This online computational method has four functional approximators: a critic approximator, two actor approximators, and a state-reward neural network (NN). It simultaneously approximates the parameters of Q -function and the learner state reward online. Convergence and stability proofs are rigorously studied to guarantee the algorithm performance.

Inverse Reinforcement Learning in Tracking Control Based on Inverse Optimal Control.

This article provides a novel inverse reinforcement learning (RL) algorithm that learns an unknown performance objective function for tracking control. The algorithm combines three steps: 1) an optimal control update; 2) a gradient descent correction step; and 3) an inverse optimal control (IOC) update. The new algorithm clarifies the relation between inverse RL and IOC. It is shown that the reward weight of an unknown performance objective that generates a target control policy may not be unique. We characterize the set of all weights that generate the same target control policy. We develop a model-based algorithm and, further, two model-free algorithms for systems with unknown model information. Finally, simulation experiments are presented to show the effectiveness of the proposed algorithms.

H∞-Based Minimal Energy Adaptive Control With Preset Convergence Rate.

This work studies the H∞ -based minimal energy control with a preset convergence rate (PCR) problem for a class of disturbed linear time-invariant continuous-time systems with matched external disturbance. This problem aims to design an optimal controller so that the energy of the control input satisfies a predetermined requirement. Moreover, the closed-loop system asymptotic stability with PCR is ensured simultaneously. To deal with this problem, a modified game algebraic Riccati equation (MGARE) is proposed, which is different from the game algebraic Riccati equation in the traditional H∞ control problem due to the state cost being lost. Therefore, a unique positive-definite solution of the MGARE is theoretically analyzed with its existing conditions. In addition, based on this formulation, a novel approach is proposed to solve the actuator magnitude saturation problem with the system dynamics being exactly known. To relax the requirement of the knowledge of system dynamics, a model-free policy iteration approach is proposed to compute the solution of this problem. Finally, the effectiveness of the proposed approaches is verified through two simulation examples.

Off-Policy Reinforcement Learning for Tracking in Continuous-Time Systems on Two Time Scales.

This article applies a singular perturbation theory to solve an optimal linear quadratic tracker problem for a continuous-time two-time-scale process. Previously, singular perturbation was applied for system regulation. It is shown that the two-time-scale tracking problem can be separated into a linear-quadratic tracker (LQT) problem for the slow system and a linear-quadratic regulator (LQR) problem for the fast system. We prove that the solutions to these two reduced-order control problems can approximate the LQT solution of the original control problem. The reduced-order slow LQT and fast LQR control problems are solved by off-policy integral reinforcement learning (IRL) using only measured data from the system. To test the effectiveness of the proposed method, we use an industrial thickening process as a simulation example and compare our method to a method with the known system model and a method without time-scale separation.

KIF5B modulates central spindle organization in late-stage cytokinesis in chondrocytes.

BACKGROUND: The growth plate is a special region of the cartilage that drives longitudinal growth of long bones. Proliferating chondrocytes in the growth plate, arranged in columns, divide perpendicular to the long axis of the growth plate then intercalate to re-align with parental columns. Which molecular partners maintain growth plate columnar structures and chondrocyte cytokinesis has not been fully revealed. It is reported that kinesin family member 3A (KIF3A), a subunit of kinesin-2, plays an important role in maintaining columnar organization in growth plates via controlling primary cilia formation and cell proliferation. RESULT: Here we identify kinesin family member 5B (KIF5B), the heavy chain of kinesin-1, a ubiquitously expressed motor protein for anterograde intracellular transport along the microtubule network, as a key modulator of cytokinesis in chondrocytes via maintenance of central spindle organization. We show that KIF5B is concentrated in the central spindle during cytokinesis in both primary chondrocytes and chondrogenic ATDC5 cells. CONCLUSION: The failure of cytokinesis in KIF5B null chondrocytes leads to incomplete cell rotation, disrupting proliferation and differentiation, and results in a disorganized growth plate.

A new role of anterograde motor Kif5b in facilitating large clathrin-coated vesicle mediated endocytosis via regulating clathrin uncoating.

Kif5b-driven anterograde transport and clathrin-mediated endocytosis (CME) are responsible for opposite intracellular trafficking, contributing to plasma membrane homeostasis. However, whether and how the two trafficking processes coordinate remain unclear. Here, we show that Kif5b directly interacts with clathrin heavy chain (CHC) at a region close to that for uncoating catalyst (Hsc70) and preferentially localizes on relatively large clathrin-coated vesicles (CCVs). Uncoating in vitro is decreased for CCVs from the cortex of kif5b conditional knockout (mutant) mouse and facilitated by adding Kif5b fragments containing CHC-binding site, while cell peripheral distribution of CHC or Hsc70 keeps unaffected by Kif5b depletion. Furthermore, cellular entry of vesicular stomatitis virus that internalizes into large CCV is inhibited by Kif5b depletion or introducing a dominant-negative Kif5b fragment. These findings showed a new role of Kif5b in regulating large CCV-mediated CME via affecting CCV uncoating, indicating Kif5b as a molecular knot connecting anterograde transport to CME.

Multimodal nonlinear optical microscopy reveals critical role of kinesin-1 in cartilage development.

We developed a multimodal nonlinear optical (NLO) microscope system by integrating stimulated Raman scattering (SRS), second harmonic generation (SHG) and two-photon excited fluorescence (TPEF) imaging. The system was used to study the morphological and biochemical characteristics of tibial cartilage in a kinesin-1 (Kif5b) knockout mouse model. The detailed structure of fibrillar collagen in the extracellular matrix of cartilage was visualized by the forward and backward SHG signals, while high resolution imaging of chondrocytes was achieved by capturing endogenous TPEF and SRS signals of the cells. The results demonstrate that collagen fibrils in the superficial surface of the articular cartilage decreased significantly in the absence of Kif5b. The distorted morphology along with accumulated intracellular collagen was observed in the Kif5b-deficient chondrocytes, indicating the critical roles of kinesin-1 in the chondrocyte morphogenesis and collagen secretion. The study shows that multimodal NLO imaging method is an effective approach to investigate early development of cartilage.

Analysis of Kif5b expression during mouse kidney development.

Recent studies showed that kidney-specific inactivation of Kif3a produces kidney cysts and renal failure, suggesting that kinesin-mediated intracellular transportation is important for the establishement and maintenance of renal epithelial cell polarity and normal nephron functions. Kif5b, one of the most conserved kinesin heavy chain, is the mouse homologue of the human ubiquitous Kinesin Heavy Chain (uKHC). In order to elucidate the role of Kif5b in kidney development and function, it is essential to establish its expression profile within the organ. Therefore, in this study, we examined the expression pattern of Kif5b in mouse kidney. Kidneys from embryonic (E) 12.5-, 16.5-dpc (days post coitus) mouse fetuses, from postnatal (P) day 0, 10, 20 pups and from adult mice were collected. The distribution of Kif5b was analyzed by immunostaining. The possible involvement of Kif5b in kidney development was investigated in conditional mutant mice by using a Cre-LoxP strategy. This study showed that the distribution of Kif5b displayed spatiotemporal changes during postnatal kidney development. In kidneys of new born mice, Kif5b was strongly expressed in all developing tubules and in the ureteric bud, but not in the glomerulus or in other early-developing structures, such as the cap mesenchyme, the comma-shaped body, and the S-shaped body. In kidneys of postnatal day 20 or of older mice, however, Kif5b was localized selectively in the basolateral domain of epithelial cells of the thick ascending loop of Henle, as well as of the distal convoluted tubule, with little expression being observed in the proximal tubule or in the collecting duct. Conditional knock-down of Kif5b in mouse kidney did not result in detectable morphological defects, but it did lead to a decrease in cell proliferation rate and also to a mislocalization of Na+/K+/-ATPase, indicating that although Kif5b is non-essential for kidney morphogenesis, it is important for nephron maturation.

Dissect Kif5b in nuclear positioning during myogenesis: the light chain binding domain and the autoinhibitory peptide are both indispensable.

The microtubule motor kinesin-1 is responsible for the nuclear positioning during myogenesis. Here we show that the coiled-coil stalk/tail domain containing the kinesin light chain (KLC) binding sites targets to the perinuclear region like endogenous Kif5b, while the globular tail domain cannot. To investigate which fragments of kinesin heavy chain (Kif5b) is responsible for the myonuclear positioning, we transfect Kif5b expression constructs into Kif5b deficient myoblasts and test their ability to rescue the myonuclear phenotype. We find that the KLC binding domain and the autoinhibitory peptide in the globular tail region are both indispensable for the nuclear membrane localization of Kif5b and the kinesin-1-mediated myonuclear positioning. These results suggest that while the KLC binding domain may directly targets Kif5b to the myonuclear membrane, the autoinhibitory peptide may play an indirect role in regulating the kinesin-1-mediated myonuclear positioning.

Kif5b controls the localization of myofibril components for their assembly and linkage to the myotendinous junctions.

Controlled delivery of myofibril components to the appropriate sites of assembly is crucial for myofibrillogenesis. Here, we show that kinesin-1 heavy chain Kif5b plays important roles in anterograde transport of α-sarcomeric actin, non-muscle myosin IIB, together with intermediate filament proteins desmin and nestin to the growing tips of the elongating myotubes. Mice with Kif5b conditionally knocked out in myogenic cells showed aggregation of actin filaments and intermediate filament proteins in the differentiating skeletal muscle cells, which further affected myofibril assembly and their linkage to the myotendinous junctions. The expression of Kif5b in mutant myotubes rescued the localization of the affected proteins. Functional mapping of Kif5b revealed a 64-amino acid α-helix domain in the tail region, which directly interacted with desmin and might be responsible for the transportation of these proteins in a complex.